feat(design): auto-retry crashed persona once before giving up#77
feat(design): auto-retry crashed persona once before giving up#77kwliang1 wants to merge 2 commits into
Conversation
sf8193
left a comment
There was a problem hiding this comment.
Reviewed with dual-agent process (sp-reviewer + typescript-reviewer). 5 should-fix, 3 nits.
| } | ||
| } catch (err) { | ||
| process.stderr.write(`daemon: design: ${name} auto-retry failed: ${err}\n`) | ||
| onDesignParticipantDisconnect(deadPersona.sessionId) |
There was a problem hiding this comment.
Should-fix (flagged by both reviewers): catch block can double-decrement expectation counters.
If doSpawnSession throws before line 642 mutates deadPersona.sessionId, this calls onDesignParticipantDisconnect(oldSessionId). The handler finds the same persona again (sessionId still matches), sees it's in retriedPersonas, and decrements questionsExpected/proposalsExpected/refinementExpected a second time for the same dead persona. This can push counters negative.
Conversely, if spawn succeeds (sessionId was mutated), this calls disconnect with the new sessionId — which works correctly (falls through to retry-exhausted path). But the asymmetry is subtle and the failure path is buggy.
Suggestion: capture the old sessionId before the try block and use it to guard against re-entry, or restructure so the catch doesn't re-enter the disconnect handler.
| }) | ||
|
|
||
| // Update session ID immediately so disconnect handler can find it if the new session crashes fast | ||
| deadPersona.sessionId = result.sessionId |
There was a problem hiding this comment.
Should-fix: in-place sessionId mutation loses the old session ID.
Two consequences:
- If the old session's process is still lingering (delayed teardown),
cleanupDesignSessionsiteratesstate.personasand only sees the new sessionId — the old process leaks. - Late-arriving disconnect events for the old sessionId won't match any persona in
state.personas, so they're silently ignored. This is benign but makes debugging harder.
Consider keeping the old ID (e.g., a previousSessionIds set on the persona) or treating the replacement as a distinct entry.
| if (state.phase === 'questioning') { | ||
| state.questionsExpected-- | ||
| if (state.questionsExpected > 0 && state.questionsReceived >= state.questionsExpected) { | ||
| if (state.questionsReceived >= state.questionsExpected) { |
There was a problem hiding this comment.
Should-fix (flagged by both reviewers): removing the > 0 guards enables premature phase transitions.
The old code had state.questionsExpected > 0 && (and similarly for proposals/refinement). With the guard removed, if the last persona dies and retry is exhausted, questionsExpected decrements to 0, and 0 >= 0 triggers the transition. The aliveCount === 0 cancellation check should catch this first — but it won't if another persona is still alive (aliveCount > 0) while this counter hits 0. That advances the phase with zero expectations from this counter.
Suggestion: restore the > 0 guards, or move the aliveCount === 0 cancellation to before the per-phase counter adjustment.
| }) | ||
| } else if (state.phase === 'refinement' && state.activeDivergence) { | ||
| if (state.activeDivergence.personas.some(n => n === name || n === name.replace('_', ' '))) { | ||
| state.refinementExpected++ |
There was a problem hiding this comment.
Should-fix: refinementExpected++ without checking if the dead persona already responded.
When the first crash triggers the retry path (line 718-719), the early return skips the refinementExpected-- that normally runs. So the count is still inflated by the dead persona's slot. Then here, refinementExpected++ adds another slot for the replacement — net +1 over what it should be. If the dead persona had already responded before crashing, the design now waits for an extra response that may never come (timeout).
Suggestion: check state.refinementRespondedIds.has(deadPersona.sessionId) or equivalent before incrementing.
| if (persona) { | ||
| process.stderr.write(`daemon: design: ${label} disconnected/died\n`) | ||
| void gateway.send(threadId, `_${label} disconnected. Continuing with ${state.personas.filter(p => p.sessionId !== sessionId).length} remaining personas._`).catch(() => {}) | ||
| const aliveCount = state.personas.filter(p => p.sessionId !== sessionId && transport.has(p.sessionId)).length |
There was a problem hiding this comment.
Should-fix: aliveCount can be stale under concurrent persona crashes.
If two personas crash near-simultaneously and both retries are exhausted, each computes aliveCount === 1 (seeing the other as alive before its transport is torn down). Neither triggers the aliveCount === 0 cancellation, and the design hangs until timeout.
This is a narrow race but possible under load. A mitigation: recheck aliveCount just before the cancel decision, or move the cancel check into a shared function that recalculates.
| refinementExpected: number | ||
| refinementResponses: number | ||
| refinementRespondedIds: Set<string> | ||
| activeDivergence?: { description: string; personas: string[]; impact: string } |
There was a problem hiding this comment.
Nit: activeDivergence is typed inline. Since refinementQueue presumably holds the same shape, consider referencing a shared Divergence type to avoid drift.
| meta: { chat_id: threadId, message_id: '', user: 'system', user_id: 'system', ts: cutoffTs }, | ||
| }) | ||
| } else if (state.phase === 'refinement' && state.activeDivergence) { | ||
| if (state.activeDivergence.personas.some(n => n === name || n === name.replace('_', ' '))) { |
There was a problem hiding this comment.
Nit (flagged by both reviewers): The name.replace('_', ' ') normalization duplicates logic from processNextDivergence. If the rule changes, both sites need updating. Consider extracting a helper or normalizing at the source (where activeDivergence.personas is populated).
| } | ||
|
|
||
| const divergence = state.refinementQueue.shift()! | ||
| state.activeDivergence = divergence |
There was a problem hiding this comment.
Nit: activeDivergence is set here but never cleared (e.g., when refinement completes or the queue is exhausted). Currently safe because retryPersona checks state.phase === 'refinement' before using it, but clearing it explicitly would prevent future bugs.
sf8193
left a comment
There was a problem hiding this comment.
Dual-agent review (sp-reviewer + typescript-reviewer). 2 blockers, 3 should-fix, 2 nits.
| }) | ||
|
|
||
| // Update session ID immediately so disconnect handler can find it if the new session crashes fast | ||
| deadPersona.sessionId = result.sessionId |
There was a problem hiding this comment.
Blocker (flagged by both reviewers): deadPersona.sessionId is mutated to the new session ID before the health check. If waitForBridge returns false, killSession runs, then the catch block calls onDesignParticipantDisconnect(deadPersona.sessionId) with the new (already-killed) session ID. The disconnect handler lookup may silently no-op if the kill already cleaned it from the registry, leaving the dead persona unaccounted for — expectations never decrement.
Fix: move this assignment after the waitForBridge success path (after line 649), or stash the original ID and pass it to the recursive onDesignParticipantDisconnect call.
| if (state.phase === 'questioning') { | ||
| state.questionsExpected-- | ||
| if (state.questionsExpected > 0 && state.questionsReceived >= state.questionsExpected) { | ||
| if (state.questionsReceived >= state.questionsExpected) { |
There was a problem hiding this comment.
Blocker (flagged by both reviewers): Removing the questionsExpected > 0 guard (and equivalently at lines 744, 755) allows 0 >= 0 to trigger a phase transition with zero contributions.
Scenario: persona A is alive and working, persona B (retry exhausted) disconnects. aliveCount > 0 so the cancellation path doesn't fire. questionsExpected decrements to 0, questionsReceived is 0, 0 >= 0 is true → phase advances with zero questions. Persona A's eventual submission arrives in the wrong phase and is silently dropped.
Fix: restore a > 0 guard (e.g. state.questionsExpected > 0 && state.questionsReceived >= state.questionsExpected), or ensure the all-dead check accounts for this edge.
| } | ||
|
|
||
| const divergence = state.refinementQueue.shift()! | ||
| state.activeDivergence = divergence |
There was a problem hiding this comment.
Should-fix (flagged by both reviewers): activeDivergence is set here but never cleared when refinement for this divergence completes or when the phase changes. If a persona crashes after refinement finishes but before the next divergence (or during synthesis), retryPersona will see the stale divergence, send a refinement prompt, and increment refinementExpected — corrupting the counter.
Fix: set state.activeDivergence = undefined at the end of processNextDivergence (when all responses are in) or on phase transitions out of refinement.
| if (!state.retriedPersonas.has(persona.name)) { | ||
| state.retriedPersonas.add(persona.name) | ||
| void gateway.send(threadId, `_${label} crashed — auto-retrying..._`).catch(() => {}) | ||
| void retryPersona(state, persona, threadId) |
There was a problem hiding this comment.
Should-fix: retryPersona is fire-and-forget (void) and takes 30+ seconds (spawn + health check). During that time, timeouts can fire and advance the phase. The phase-specific prompt selection (lines 661-683) uses the phase at completion time, not at crash time — so a persona that crashed during questioning could receive an independent or refinement prompt if the phase advanced while retry was in-flight.
Also, if cancelDesign runs concurrently (user cancel or all-dead from another disconnect), the retry can finish spawning a session that cancel already iterated past, leaving an orphan.
Consider: (1) capture the phase at crash time and use it for prompt selection, (2) check phase before spawn in addition to after.
| }) | ||
| } else if (state.phase === 'refinement' && state.activeDivergence) { | ||
| if (state.activeDivergence.personas.some(n => n === name || n === name.replace('_', ' '))) { | ||
| state.refinementExpected++ |
There was a problem hiding this comment.
Should-fix: If between the persona's death (which decremented refinementExpected) and the retry completing here, refinementResponses >= refinementExpected became true and processNextDivergence already advanced, then this increment makes refinementExpected permanently higher than the number of responses that will arrive — the divergence counter is off.
| refinementExpected: number | ||
| refinementResponses: number | ||
| refinementRespondedIds: Set<string> | ||
| activeDivergence?: { description: string; personas: string[]; impact: string } |
There was a problem hiding this comment.
Nit: This inline type { description: string; personas: string[]; impact: string } is presumably the same shape as divergence objects in refinementQueue. Extract a named type to keep them in sync.
| meta: { chat_id: threadId, message_id: '', user: 'system', user_id: 'system', ts: cutoffTs }, | ||
| }) | ||
| } else if (state.phase === 'refinement' && state.activeDivergence) { | ||
| if (state.activeDivergence.personas.some(n => n === name || n === name.replace('_', ' '))) { |
There was a problem hiding this comment.
Nit: String.replace with a string pattern only replaces the first occurrence. If persona names can contain multiple underscores (e.g. ux_design_lead), this won't match. Consider replaceAll('_', ' ') or a regex /_ /g. (Pre-existing at line 584 too.)
When all design personas crash, the system previously waited 5–20 minutes for timeouts before cancelling. Now it detects the all-dead state on each disconnect and cancels immediately with a clear retry message. Also fixes the "Continuing with N remaining personas" count — it was always showing total-1 because it only excluded the currently disconnecting persona, not previously dead ones. Now counts only personas with a live transport connection. Removes the `> 0` guard on questionsExpected/proposalsExpected/ refinementExpected checks — since the all-dead early return handles the zero case, these guards were preventing legitimate "last persona disconnected but some had already submitted" from advancing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
6057692 to
b690e67
Compare
| promptBuilder: (sessionId, tmuxName) => designPersonaPrompt({ sessionId, tmuxName, persona: name, topic: state.topic, threadId, cutoffTs }), | ||
| }) | ||
|
|
||
| deadPersona.sessionId = result.sessionId |
There was a problem hiding this comment.
Blocker (bump): deadPersona.sessionId is mutated to the new session ID before waitForBridge. If the health check fails, killSession runs on the new session, then the catch calls onDesignParticipantDisconnect(deadPersona.sessionId) with the already-killed new ID — the disconnect handler may no-op (registry cleaned), leaving counters uncorrected.
Fix: move this assignment after the waitForBridge success path, or stash the original ID and use it in the catch.
| if (state.phase === 'questioning') { | ||
| state.questionsExpected-- | ||
| if (state.questionsExpected > 0 && state.questionsReceived >= state.questionsExpected) { | ||
| if (state.questionsReceived >= state.questionsExpected) { |
There was a problem hiding this comment.
Blocker (bump): Removing the questionsExpected > 0 guard (and equivalently for proposals/refinement at lines 744, 755) allows 0 >= 0 to trigger a phase transition with zero contributions.
This is reachable when combined with the counter-corruption race from the retryingPersonas finding below: a delayed disconnect can decrement counters while a retry is in-flight, pushing *Expected to 0 while a live persona hasn't yet submitted. Restoring the > 0 guard acts as a safety net against this edge case.
| if (persona) { | ||
| process.stderr.write(`daemon: design: ${label} disconnected/died\n`) | ||
| void gateway.send(threadId, `_${label} disconnected. Continuing with ${state.personas.filter(p => p.sessionId !== sessionId).length} remaining personas._`).catch(() => {}) | ||
| const aliveCount = state.personas.filter(p => p.sessionId !== sessionId && transport.has(p.sessionId)).length |
There was a problem hiding this comment.
Should-fix (bump): aliveCount is computed once at the top but used later for the cancel decision. If two personas crash near-simultaneously and both retries are exhausted, each sees the other as alive before its transport tears down — neither triggers aliveCount === 0, and the design hangs until timeout.
Mitigation: recheck alive count just before the cancel decision.
| if (!state.retriedPersonas.has(persona.name)) { | ||
| state.retriedPersonas.add(persona.name) | ||
| void gateway.send(threadId, `_${label} crashed — auto-retrying..._`).catch(() => {}) | ||
| void retryPersona(state, persona, threadId) |
There was a problem hiding this comment.
Should-fix (bump): retryPersona is fire-and-forget and takes ~30s (spawn + health check). During that window, timeouts can fire and advance the phase. The phase-specific prompt at completion time may not match the phase at crash time.
Consider: (1) capture the phase at crash time for prompt selection, (2) check phase before spawn in addition to after.
| const aliveCount = state.personas.filter(p => p.sessionId !== sessionId && transport.has(p.sessionId)).length | ||
| process.stderr.write(`daemon: design: ${label} disconnected/died (${aliveCount} alive)\n`) | ||
|
|
||
| if (!state.retriedPersonas.has(persona.name)) { |
There was a problem hiding this comment.
Should-fix (new, flagged by both reviewers): retryingPersonas is written to (line 635 add, line 687 delete) but never read anywhere. If a delayed disconnect event for the old sessionId arrives while retryPersona is in-flight, the handler finds the persona, sees retriedPersonas.has(name) is true, and falls through to the retry-exhausted path — decrementing counters even though the retry might succeed.
Add state.retryingPersonas.has(persona.name) as a guard here to skip counter adjustment while a retry is in-flight. Otherwise the set is dead code and should be removed.
| if (aliveCount === 0) { | ||
| if (state.timeout) { clearTimeout(state.timeout); state.timeout = undefined } | ||
| process.stderr.write(`daemon: design: all personas dead — cancelling\n`) | ||
| void cancelDesign(threadId, `All personas crashed. Design cancelled.\nUse \`design: ${state.topic}\` to retry.`).catch(e => process.stderr.write(`daemon: design: cancel failed: ${e}\n`)) |
There was a problem hiding this comment.
Should-fix (new, flagged by both reviewers): cancelDesign is async and fire-and-forget here. Between the void call and cancelDesign setting state.phase = 'cancelled', another disconnect event could enter the handler, see aliveCount === 0 again, and fire a second concurrent cancel.
Fix: set state.phase = 'cancelled' synchronously before the void cancelDesign(...) call:
state.phase = 'cancelled' as any // prevent re-entry
void cancelDesign(threadId, ...).catch(...)| refinementExpected: number | ||
| refinementResponses: number | ||
| refinementRespondedIds: Set<string> | ||
| activeDivergence?: { description: string; personas: string[]; impact: string } |
There was a problem hiding this comment.
Nit (bump): activeDivergence inline type { description: string; personas: string[]; impact: string } is the same shape as divergence objects in refinementQueue. Extract a shared Divergence type to prevent drift.
|
Wait why do we need this? Personas shouldn't be crashing |
b690e67 to
b92fa53
Compare
| promptBuilder: (sessionId, tmuxName) => designPersonaPrompt({ sessionId, tmuxName, persona: name, topic: state.topic, threadId, cutoffTs }), | ||
| }) | ||
|
|
||
| deadPersona.sessionId = result.sessionId |
There was a problem hiding this comment.
Blocker (bump, flagged by both reviewers): deadPersona.sessionId is mutated to the new session ID before waitForBridge. If the health check fails, killSession runs on the new session, then the catch calls onDesignParticipantDisconnect(deadPersona.sessionId) — but that's now the new session ID, not the old one. The old session's tmux/registry entry is never cleaned up (resource leak), and the disconnect handler looks up the wrong persona.
Additionally, the old session is never explicitly killed before spawning the replacement. Compare with the synthesizer timeout handler which kills the old session before respawning.
Fix: save the old session ID, kill the old session before spawning, and only assign the new session ID after the health check succeeds.
| if (state.phase === 'questioning') { | ||
| state.questionsExpected-- | ||
| if (state.questionsExpected > 0 && state.questionsReceived >= state.questionsExpected) { | ||
| if (state.questionsReceived >= state.questionsExpected) { |
There was a problem hiding this comment.
Blocker (bump, flagged by both reviewers): Removing the questionsExpected > 0 guard (and equivalently at lines 747, 758) allows 0 >= 0 to trigger a phase transition with zero contributions.
Scenario: 2 personas alive, both crash in the same phase. First crash triggers retry (returns early). Second crash hits the exhausted path, decrements expected to 0. 0 >= 0 is true → advances to the next phase with zero data.
The aliveCount === 0 check at line 728 only fires when retries are exhausted. If one persona is retrying and the other dies with retry exhausted, aliveCount may be 0 but the code reaches the phase-advance path. Restore the > 0 guard or add an explicit zero-expected check.
| void gateway.send(threadId, `_${label} disconnected. Continuing with ${state.personas.filter(p => p.sessionId !== sessionId).length} remaining personas._`).catch(() => {}) | ||
| if (state.retryingPersonas.has(persona.name)) return | ||
|
|
||
| const aliveCount = state.personas.filter(p => p.sessionId !== sessionId && transport.has(p.sessionId)).length |
There was a problem hiding this comment.
Should-fix (bump): aliveCount is computed once at the top but used later for the cancel decision. If two personas crash near-simultaneously and both retries are exhausted, each computes aliveCount === 1 (seeing the other as alive), so neither triggers the all-dead cancellation. Result: design stuck with no live personas.
Consider recomputing aliveCount at the point of use, or using an atomic decrement-and-check pattern.
| if (!state.retriedPersonas.has(persona.name)) { | ||
| state.retriedPersonas.add(persona.name) | ||
| void gateway.send(threadId, `_${label} crashed — auto-retrying..._`).catch(() => {}) | ||
| void retryPersona(state, persona, threadId) |
There was a problem hiding this comment.
Should-fix (bump, flagged by both reviewers): retryPersona is fire-and-forget (void) and takes ~30s (spawn + health check). During that window, timeouts can fire and advance the phase. The phase-specific prompt selection at the end of retryPersona (lines 659-680) reads state.phase which may have changed by then. The retried persona gets instructions for a phase that already ended.
Consider capturing the phase at retry start and comparing when the retry completes, or adding a phase-change check before sending the prompt.
| if (aliveCount === 0) { | ||
| if (state.timeout) { clearTimeout(state.timeout); state.timeout = undefined } | ||
| process.stderr.write(`daemon: design: all personas dead — cancelling\n`) | ||
| void cancelDesign(threadId, `All personas crashed. Design cancelled.\nUse \`design: ${state.topic}\` to retry.`).catch(e => process.stderr.write(`daemon: design: cancel failed: ${e}\n`)) |
There was a problem hiding this comment.
Should-fix (bump): cancelDesign is async and fire-and-forget here. Between the void call and cancelDesign setting state.phase = 'cancelled', another disconnect event could enter onDesignParticipantDisconnect, pass the designs.get(threadId) check, and attempt its own cancel — causing double cleanup.
Consider setting state.phase = 'cancelled' synchronously before awaiting cancelDesign, or adding a guard at the top of cancelDesign for the cancelled phase.
| } | ||
| } catch (err) { | ||
| process.stderr.write(`daemon: design: ${name} auto-retry failed: ${err}\n`) | ||
| onDesignParticipantDisconnect(deadPersona.sessionId) |
There was a problem hiding this comment.
Should-fix (new, flagged by ts-reviewer): When the retry fails, catch calls onDesignParticipantDisconnect(deadPersona.sessionId). But at that point retryingPersonas still has the persona name (the finally block hasn't run yet). The disconnect handler hits if (state.retryingPersonas.has(persona.name)) return and exits silently — no "retry exhausted" message, no expectation adjustment, no aliveCount === 0 check. The persona just vanishes.
Fix: delete from retryingPersonas before calling onDesignParticipantDisconnect in the catch block:
} catch (err) {
process.stderr.write(`daemon: design: ${name} auto-retry failed: ${err}\n`)
state.retryingPersonas.delete(name)
onDesignParticipantDisconnect(deadPersona.sessionId)
}And remove the delete from finally (or keep it as a no-op safety net).
| state.retryingPersonas.add(name) | ||
|
|
||
| try { | ||
| const result = await doSpawnSession(`Design persona: ${name}`, undefined, undefined, { |
There was a problem hiding this comment.
Should-fix (new, flagged by both reviewers): The old persona session is never killed before spawning the replacement. The bridge disconnected (that's how we got here), but the tmux session and registry entry for the old sessionId may linger. This leaks resources.
Add explicit cleanup before spawning:
const oldInfo = registry.get(deadPersona.sessionId)
if (oldInfo && !killsInProgress.has(deadPersona.sessionId)) {
await killSession(oldInfo, 'persona crashed, retrying').catch(() => {})
}| refinementExpected: number | ||
| refinementResponses: number | ||
| refinementRespondedIds: Set<string> | ||
| activeDivergence?: { description: string; personas: string[]; impact: string } |
There was a problem hiding this comment.
Nit (bump): activeDivergence inline type { description: string; personas: string[]; impact: string } is the same shape as divergence objects in refinementQueue. Extract a shared Divergence type to keep them in sync.
b92fa53 to
9354eed
Compare
| promptBuilder: (sessionId, tmuxName) => designPersonaPrompt({ sessionId, tmuxName, persona: name, topic: state.topic, threadId, cutoffTs }), | ||
| }) | ||
|
|
||
| deadPersona.sessionId = result.sessionId |
There was a problem hiding this comment.
Blocker (bump, flagged by both reviewers): deadPersona.sessionId is mutated to the new session ID before waitForBridge. If the health check fails, killSession runs on the new session, then the catch at line 690 deletes retryingPersonas and calls onDesignParticipantDisconnect(deadPersona.sessionId) — but that's now the new (already-killed) session ID, not the original dead one. The disconnect handler either finds no matching persona or finds the stale entry, leading to expectations never being decremented and the phase getting stuck.
Acknowledged: old-session cleanup (line 638-641) and retryingPersonas guard (line 722) were added to address prior feedback — those are good. The core issue is the mutation ordering.
Fix: capture const originalSessionId = deadPersona.sessionId before line 643 and use it in the catch block. Move the mutation to after the health check succeeds.
| if (state.questionsExpected > 0 && state.questionsReceived >= state.questionsExpected) { | ||
| if (state.questionsReceived >= state.questionsExpected) { | ||
| if (state.timeout) clearTimeout(state.timeout) | ||
| const result = designMachine.transition(state.phase, 'all_questions') |
There was a problem hiding this comment.
Blocker (bump, flagged by both reviewers): Removing the questionsExpected > 0 guard (and equivalently at lines 752, 763) allows 0 >= 0 to trigger a phase transition with zero contributions.
Scenario: 2 personas, both crash in questioning phase. First crash triggers retry (returns early, no decrement). Second crash: retry exhausted → questionsExpected-- → now 0. 0 >= 0 is true → fires aggregateAndPostQuestions with an empty question set. The aliveCount === 0 cancel path fires cancelDesign with void (async), so the synchronous counter check runs first.
Fix: restore > 0 guards, or add if (aliveCount === 0) return before the counter-decrement blocks (the cancel already handles this case).
|
|
||
| const aliveCount = state.personas.filter(p => p.sessionId !== sessionId && transport.has(p.sessionId)).length | ||
| process.stderr.write(`daemon: design: ${label} disconnected/died (${aliveCount} alive)\n`) | ||
|
|
There was a problem hiding this comment.
Should-fix (bump): aliveCount is computed once at line 724 but used later for the cancel decision at line 736. If two personas crash near-simultaneously and both retries are exhausted, each computes aliveCount === 1 (seeing the other as alive before its transport tears down) — neither triggers aliveCount === 0, and the design hangs with no personas.
Fix: recompute aliveCount at the cancel decision point, or set state.phase = 'cancelled' synchronously before the void cancelDesign(...) call so subsequent disconnect events short-circuit.
| void gateway.send(threadId, `_${label} crashed — auto-retrying..._`).catch(() => {}) | ||
| void retryPersona(state, persona, threadId) | ||
| return | ||
| } |
There was a problem hiding this comment.
Should-fix (bump, flagged by both reviewers): retryPersona is fire-and-forget (void) and takes ~30s (spawn + health check). During that window, the phase timeout can fire and advance the phase. The phase-specific prompt at the end of retryPersona (lines 667-687) reads state.phase at completion time, not at crash time — so a persona that crashed during independent could receive a refinement prompt (or no prompt at all if the phase doesn't match).
Fix: capture const crashPhase = state.phase at call time and use it for prompt selection inside retryPersona.
| process.stderr.write(`daemon: design: all personas dead — cancelling\n`) | ||
| void cancelDesign(threadId, `All personas crashed. Design cancelled.\nUse \`design: ${state.topic}\` to retry.`).catch(e => process.stderr.write(`daemon: design: cancel failed: ${e}\n`)) | ||
| return | ||
| } |
There was a problem hiding this comment.
Should-fix (bump): cancelDesign is async and fire-and-forget here. Between the void call and cancelDesign setting state.phase = 'cancelled', another disconnect event can enter the handler, see aliveCount === 0 again, and fire a second concurrent cancel — double cleanup, double user message.
Fix: set state.phase = 'cancelled' synchronously before void cancelDesign(...), so subsequent disconnect events see the cancelled state and bail early.
| ): Promise<void> { | ||
| const name = deadPersona.name as PersonaName | ||
| const cutoffTs = new Date().toISOString() | ||
| state.retryingPersonas.add(name) |
There was a problem hiding this comment.
Should-fix (new, flagged by ts-reviewer): No early-exit guard for state.phase === 'cancelled' || 'complete' at the top of retryPersona before calling doSpawnSession. If the design is cancelled (e.g., by the user) while the persona crash is being processed, the retry still proceeds to spawn a new session, wait up to HEALTH_TIMEOUT_MS, and only then clean it up at line 658-662.
Fix: add if (state.phase === 'cancelled' || state.phase === 'complete') return before line 637.
| }) | ||
| } else if (state.phase === 'refinement' && state.activeDivergence) { | ||
| if (state.activeDivergence.personas.some(n => n === name || n === name.replaceAll('_', ' '))) { | ||
| transport.sendOrQueue(result.sessionId, { |
There was a problem hiding this comment.
Nit (new, flagged by both reviewers): replaceAll('_', ' ') here but processNextDivergence (pre-existing, line ~584) uses replace('_', ' ') which only replaces the first occurrence. If a persona name has multiple underscores (e.g. ux_design_lead), the match logic diverges. Both sites should use the same normalization — ideally extract a shared personaNameMatches(a, b) helper.
| refinementExpected: number | ||
| refinementResponses: number | ||
| refinementRespondedIds: Set<string> | ||
| activeDivergence?: { description: string; personas: string[]; impact: string } |
There was a problem hiding this comment.
Nit (bump): This inline type is the same shape as divergence objects in refinementQueue. Extract a shared Divergence type to prevent drift.
When a design persona crashes, the system now automatically respawns it once before decrementing expected counts. Key design decisions: - `retriedPersonas` Set tracks which personas have used their one auto-retry, preventing infinite loops - `retryingPersonas` Set tracks in-flight retries, preventing the manual `retry design` command from double-spawning - `activeDivergence` stored in state (set in processNextDivergence, cleared when queue empties) so retried personas get correct refinement context - Auto-retry fills the SAME slot — no refinementExpected++ needed since the disconnect handler returns early without decrementing - Session ID updated before waitForBridge so fast-crash disconnect handler can find the persona - Post-spawn phase check kills new session if design was cancelled during the async retry window Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
9354eed to
62a9f36
Compare
| joinThread: threadId, | ||
| memberLabel: name, | ||
| promptBuilder: (sessionId, tmuxName) => designPersonaPrompt({ sessionId, tmuxName, persona: name, topic: state.topic, threadId, cutoffTs }), | ||
| }) |
There was a problem hiding this comment.
Blocker (bump, flagged by both reviewers): deadPersona.sessionId is mutated to the new session ID before waitForBridge. If the health check fails, killSession runs on the new session, then the catch calls onDesignParticipantDisconnect(deadPersona.sessionId) — which is now the new ID. The fallback path works by accident (the new session was killed so transport.has returns false), but the mutation-then-throw pattern is fragile.
Fix: Defer mutation until after health check:
const newSessionId = result.sessionId
const ok = await waitForBridge(newSessionId, HEALTH_TIMEOUT_MS)
if (!ok) { /* kill newSessionId, throw */ }
deadPersona.sessionId = newSessionId // only now| if (state.phase === 'questioning') { | ||
| state.questionsExpected-- | ||
| if (state.questionsExpected > 0 && state.questionsReceived >= state.questionsExpected) { | ||
| if (state.questionsReceived >= state.questionsExpected) { |
There was a problem hiding this comment.
Blocker (bump, flagged by both reviewers): Removing the questionsExpected > 0 guard (and equivalently at lines 756, 767) allows 0 >= 0 to trigger a phase transition with zero contributions.
The aliveCount === 0 cancel path at line 737 is meant to replace this guard, but under concurrent persona crashes + retry failures, aliveCount can be stale (see line 725 comment). If the cancel path doesn't fire, the 0 >= 0 fallthrough advances the phase with no participants.
Fix: Either restore the > 0 guards as belt-and-suspenders, or add an assertion documenting the invariant that this code is only reachable when aliveCount > 0.
| void gateway.send(threadId, `_${label} disconnected. Continuing with ${state.personas.filter(p => p.sessionId !== sessionId).length} remaining personas._`).catch(() => {}) | ||
| if (state.retryingPersonas.has(persona.name)) return | ||
|
|
||
| const aliveCount = state.personas.filter(p => p.sessionId !== sessionId && transport.has(p.sessionId)).length |
There was a problem hiding this comment.
Should-fix (bump): aliveCount is computed once here but used later for the cancel decision at line 737. If two personas crash near-simultaneously and both retries are exhausted, each computes aliveCount === 1 (seeing the other as alive), both skip the cancel path, and both decrement *Expected — potentially advancing the phase with zero participants.
Additionally, aliveCount doesn't exclude personas in retryingPersonas whose old sessions are dead but new sessions aren't in transport yet — this can cause premature "all personas dead" cancellation while a retry is in flight.
Fix: Recompute alive count at the cancel decision point, and exclude retryingPersonas from the dead count.
| if (!state.retriedPersonas.has(persona.name)) { | ||
| state.retriedPersonas.add(persona.name) | ||
| void gateway.send(threadId, `_${label} crashed — auto-retrying..._`).catch(() => {}) | ||
| void retryPersona(state, persona, threadId) |
There was a problem hiding this comment.
Should-fix (bump): retryPersona is fire-and-forget (void) and takes ~30s (spawn + health check). During that window, timeouts can fire and advance the phase. The phase-specific prompt at completion time (lines 668-687) reads state.phase and state.activeDivergence from live state — if the phase advanced while the retry was in flight, the persona gets a stale or wrong prompt.
| if (aliveCount === 0) { | ||
| if (state.timeout) { clearTimeout(state.timeout); state.timeout = undefined } | ||
| process.stderr.write(`daemon: design: all personas dead — cancelling\n`) | ||
| void cancelDesign(threadId, `All personas crashed. Design cancelled.\nUse \`design: ${state.topic}\` to retry.`).catch(e => process.stderr.write(`daemon: design: cancel failed: ${e}\n`)) |
There was a problem hiding this comment.
Should-fix (bump): cancelDesign is async and fire-and-forget here. Between the void call and cancelDesign setting state.phase = 'cancelled', another disconnect event could enter onDesignParticipantDisconnect and also trigger cancel — double cleanup. Consider setting state.phase = 'cancelled' synchronously here before the void cancelDesign(...) call.
| transport.sendOrQueue(result.sessionId, { | ||
| type: 'notification', | ||
| content: `[system] You are replacing a crashed persona. Read the thread for context, then post your proposal. Tag with \`[${name}→thread]\`. Be INDEPENDENT.`, | ||
| meta: { chat_id: threadId, message_id: '', user: 'system', user_id: 'system', ts: cutoffTs }, |
There was a problem hiding this comment.
Should-fix (new, flagged by both reviewers): When a retried persona re-enters refinement, refinementExpected is not re-incremented. The disconnect handler already decremented it (line 766). If the retried persona responds, refinementResponses increases but the bookkeeping is off — the system may advance to the next divergence prematurely or count the response as unexpected.
Same issue in the independent phase (line 668): proposalsExpected was decremented by the disconnect handler but isn't re-incremented when the retry succeeds.
Fix: Re-increment refinementExpected / proposalsExpected when the retry succeeds and the persona is prompted.
| } | ||
|
|
||
| if (state.phase === 'cancelled' || state.phase === 'complete') { | ||
| const info = registry.get(result.sessionId) |
There was a problem hiding this comment.
Should-fix (new): If the design ends during retry (phase becomes cancelled/complete), this early return skips the state.retryingPersonas.delete(name) at line 694. The persona name remains permanently in retryingPersonas, which blocks future disconnect handling for that persona name.
| @@ -59,6 +59,9 @@ export type DesignState = { | |||
| refinementExpected: number | |||
| refinementResponses: number | |||
| refinementRespondedIds: Set<string> | |||
There was a problem hiding this comment.
Nit (bump): activeDivergence inline type { description: string; personas: string[]; impact: string } is presumably the same shape as divergence objects in refinementQueue. Extract a shared Divergence type to keep them in sync.
| type: 'notification', | ||
| content: `[system] You are replacing a crashed persona. Read the thread for context, then post your proposal. Tag with \`[${name}→thread]\`. Be INDEPENDENT.`, | ||
| meta: { chat_id: threadId, message_id: '', user: 'system', user_id: 'system', ts: cutoffTs }, | ||
| }) |
There was a problem hiding this comment.
Nit (new, flagged by both reviewers): replaceAll('_', ' ') here vs replace('_', ' ') (first occurrence only) in processNextDivergence. Persona names with multiple underscores (e.g. ux_design_lead) would match here but not there. Consider extracting a matchesPersonaName(a, b) helper for consistency.
So this is due to the work where I was moving the |
|
Closing — the root cause (design personas crashing when invoked from main thread) was resolved by the templates system. PR #76 (detect-all-dead + cancel immediately) provides sufficient defense-in-depth. |
Summary
Set<string>to ensure at most one auto-retry per personadeadPersona.sessionIdimmediately after spawn (beforewaitForBridge) to prevent fast-crash race conditionactiveDivergencefield toDesignStateto track the current divergence being discussedindependent, divergence context duringrefinementStack: merge #76 first.
Test plan
bun test(268 pass, 0 fail)🤖 Generated with Claude Code